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Highlights 


Why  GAO  Did  This  Study 

In  2008,  the  cumulative  cost 
growth  in  the  Department  of 
Defense’s  portfolio  of  96  major 
defense  acquisition  programs  was 
$296  billion  and  the  average  delay 
in  delivering  promised 
capabilities  to  the  warfighter  was 
22  months.  These  poor  outcomes 
mean  that  other  critical  defense 
and  national  priorities  may  go 
unfunded  and  that  warfighters 
may  go  without  the  equipment 
they  need  to  counter  the  changing 
threats  that  they  face. 

GAO  has  examined  extensively 
the  issues  in  DOD’s  weapon 
system  programs  from  the 
perspective  of  best  practices  in 
product  development,  and 
believes  that  significant 
improvements  are  possible. 
Because  the  ability  to  measure 
knowledge,  processes,  and 
outcomes  is  critical  to  achieving 
these  improvements,  GAO  has 
used  metrics  to  review  the 
management  and  health  of  these 
programs  from  within  the 
framework  of  best  practices. 

This  testimony  discusses:  1) 
“knowledge  metrics,”  used  to 
determine  how  well  programs 
manage  technology,  design,  and 
manufacturing  risks;  2)  outcome 
metrics-concerning  cost, 
schedule,  and  capability-that 
serve  as  “health  indicators”  of 
how  well  programs  are  being 
executed  in  terms  of  predicted 
outcomes;  and  3)  the 
prerequisites  that  GAO  believes 
must  be  met  in  order  for  a 
program’s  plans  and  goals  to  be 
realistic. 


View  GAO-09-543T  or  key  components. 
For  more  information,  contact  Michael  J. 
Sullivan  at  (202)  512-4841  or 
suliivanm@gao.gov. 


What  GAO  Found 

GAO  employs  a  set  of  knowledge  metrics  to  determine  whether  programs 
have  attained  the  right  knowledge  at  critical  points  over  the  course  of  a 
weapon  system  acquisition,  and  facilitate  the  identification  of  potential 
problems  that  could  lead  to  cost,  schedule,  or  performance  shortfalls.  In 
essence,  knowledge  supplants  risk  over  time.  Key  knowledge  points  and 
metrics  include  1)  achieving  a  high  level  of  technology  maturity  at  the  start  of 
program  development,  2)  reaching  design  stability  at  the  system-level  critical 
design  review,  and  3)  demonstrating  that  critical  manufacturing  processes  are 
in  control  before  starting  production.  By  applying  these  metrics  to  selected 
programs  in  DOD’s  2008  portfolio  of  major  defense  acquisitions,  GAO  found 
that  most  programs  have  started  system  development  without  mature 
technologies  and  moved  into  system  demonstration  with  low  levels  of  design 
stability.  GAO  has  determined  that  programs  with  immature  technologies  and 
unstable  designs  have  experienced  significant  cost  and  schedule  growth. 

Program  outcome  metrics — quantitative  measures  of  cost,  schedule,  and 
performance  over  time — provide  useful  indicators  of  the  health  of  acquisition 
programs  and  whether  they  are  meeting  their  intended  goals.  When  assessed 
regularly  for  changes  and  the  reasons  that  cause  changes,  these  indicators  can 
be  valuable  tools  for  improving  insight  into  and  oversight  of  individual 
programs  as  well  as  DOD’s  total  portfolio  of  major  defense  acquisitions.  The 
collective  performance  of  the  programs  in  DOD’s  portfolio  is  a  key  indicator 
of  how  well  the  acquisition  system  generates  the  return  on  investment  that  it 
promises  to  the  warfighter,  Congress  and  taxpayers.  GAO  recently  reported 
that  outcome  metrics  for  DOD’s  2008  major  defense  acquisition  portfolio 
show  worsening  performance  when  compared  to  the  department’s  2003 
portfolio.  For  example,  total  acquisition  costs  for  programs  in  the  2008 
portfolio  increased  25  percent  from  first  estimates  compared  to  a  19-percent 
increase  for  programs  in  the  2003  portfolio.  DOD  is  working  with  GAO  and  the 
Office  of  Management  and  Budget  to  develop  a  comprehensive  set  of  outcome 
metrics  to  better  assess  its  portfolio  of  programs. 

While  knowledge  and  outcome  metrics  provide  valuable  information  about 
the  potential  problems  and  health  of  programs,  they  are  of  limited  value  if 
DOD  does  not  do  a  better  job  ensuring  acquisitions  begin  with  realistic  plans 
and  baselines  prior  to  development  start.  GAO  believes  there  is  a  clear  set  of 
prerequisites  that  must  be  met  by  each  program’s  acquisition  strategy  before  a 
measurement  of  the  program’s  health  will  be  of  real  value.  These  prerequisites 
include:  1)  establishing  an  evolutionary,  knowledge-based  business  case  for 
each  acquisition;  2)  separating  technology  development  from  product 
development;  3)  limiting  time  and  requirements  for  product  development  to 
manageable  levels;  4)  employing  systems  engineering  early  on  in  the  process 
to  arrive  at  realistic  cost  and  schedule  estimates;  5)  committing  to  fully 
funding  a  program  once  it  is  approved;  and  6)  setting  priorities  from  the  top  to 
ensure  that  candidate  programs  are  truly  needed  and  have  a  solid  plan  for 
delivery. 


United  States  Government  Accountability  Office 


Mr.  Chairman  and  Members  of  the  Committee: 


I  am  pleased  to  be  here  this  morning  to  discuss  how  best  to  measure  and 
determine  whether  DOD’s  acquisition  system  is  providing  value  to  the 
warfighter.  Earlier  this  week,  we  reported  that  the  cumulative  cost  growth 
in  DOD’s  portfolio  of  96  major  defense  acquisition  programs  was  $296 
billion  and  the  average  delay  in  delivering  promised  capabilities  to  the 
warfighter  was  22  months.  These  outcomes  mean  that  other  critical 
defense  and  national  priorities  go  unfunded  and  warfighters  go  without 
the  equipment  they  need  to  counter  ever  changing  threats  that  they  face. 
This  condition  is  unacceptable.  We  believe  that  significant  improvement  in 
the  acquisition  of  weapon  systems  is  possible  and  that  the  ability  to 
measure  knowledge,  processes,  and  outcomes  is  critical  to  achieving  that 
improvement.  It  is  important  to  note  that  not  one  single  metric  or  set  of 
metrics  is  enough  to  monitor  acquisitions  and  gain  efficiencies.  Today,  we 
would  like  to  break  our  discussion  about  how  to  measure  the  department’s 
acquisitions  into  3  basic  sections: 

•  First,  we  would  like  to  present  a  set  of  metrics  that  we  refer  to  as 
“knowledge  metrics”  and  use  to  determine  how  well  acquisition  programs 
are  managing  and  retiring  predictable  technology,  design,  and 
manufacturing  risks  and  gaining  knowledge.  These  metrics  are  valuable 
because  they  can  predict  problems  and  identify  causes. 

•  Second,  we  would  like  to  discuss  a  set  of  outcome  measures — concerning 
cost,  schedule,  and  capability-that  serve  as  health  indicators.  These 
indicators  measure  how  well  programs  are  being  executed  and  achieving 
predicted  outcomes  in  terms  of  meeting  original  baselines  for  cost, 
schedule,  and  performance.  These  metrics  have  intrinsic  value  as  simple 
measurements,  just  as  a  thermometer  can  warn  a  parent  that  a  child  has  a 
fever. 

•  Third,  there  are  certain  indicators  that  we  look  for — based  on  the  work  we 
have  done  examining  best  practices  for  product  development — that  are, 
perhaps,  more  important  than  these  knowledge  and  health  metrics 
because  they  determine  from  the  outset  how  realistic  the  acquisition  plans 
and  strategies  of  programs  are.  For  the  sake  of  today’s  discussion,  we  will 
refer  to  them  as  “prerequisite  indicators.”  These  prerequisites  are  most 
important  because  we  question  the  value  of  ANY  metric  when  measuring 
from  an  unrealistic  baseline. 

We  know  that  the  knowledge  and  program  health  metrics  we  use  to 
measure  programs’  progress  and  outcomes  are  valuable  when  used  in 
realistic,  market-driven  product  development  environments.  We  also  know 
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that  ALL  of  these  metrics  are  important  indicators  for  decision  makers. 

Our  extensive  body  of  work  examining  world-class  enterprises  and  the 
way  they  operate  has  validated  their  value  for  programs  that  must  deliver  a 
new  product  to  market  at  a  certain  time  and  within  a  certain  investment 
cost  or  suffer  significant  consequences.  These  metrics  work  because  they 
are  measuring  realistic  plans  and  goals  that  are  supported  by  doable 
requirements,  accurate  cost  and  schedule  estimates,  and  stable  funding. 
The  company  developing  the  products  suffers  dire  consequences,  such  as 
loss  of  market  share,  if  these  programs  do  not  succeed. 

This  statement  draws  from  our  extensive  body  of  work  on  DOD’s 
acquisition  of  weapon  systems.  A  list  of  our  key  products  is  provided  at 
the  end  of  this  statement.  This  work  was  conducted  in  accordance  with 
generally  accepted  government  auditing  standards.  Those  standards 
require  that  we  plan  and  perform  the  audit  to  obtain  sufficient,  appropriate 
evidence  to  provide  a  reasonable  basis  for  our  findings  and  conclusions 
based  on  our  audit  objectives.  We  believe  that  the  evidence  obtained 
provides  a  reasonable  basis  for  our  findings  and  conclusions  based  on  our 
audit  objectives. 


“Knowledge”  Metrics 
Identify  Potential 
Problems  and  Their 
Likely  Causes 


We  have  conducted  a  body  of  work  that  examines  weapon  acquisition 
issues  from  a  perspective  that  draws  upon  lessons  learned  from  best 
practices  in  product  development.  Collectively,  these  practices  comprise  a 
process  that  is  anchored  in  knowledge.  Achieving  the  right  knowledge  at 
the  right  time  enables  leadership  to  make  informed  decisions  about  when 
and  how  best  to  move  into  various  expensive  acquisition  phases.  In 
essence,  knowledge  supplants  risk  over  time.  This  building  of  knowledge 
consists  of  gathering  information  about  technology,  design,  and 
manufacturing  at  three  critical  points  over  the  course  of  a  weapon  system 
program  (Figure  1).  We  have  developed  valuable  “knowledge  metrics”  that 
measure  this  knowledge  build  and  allow  us  to  identify  potential  problems 
that  could  lead  to  cost,  schedule,  or  performance  shortfalls  and  their  likely 
causes.  The  metrics  can  be  described  as 


•  Knowledge  Point  1 ,  evidenced  by  the  balance  between  a  product’s 
required  capabilities  and  the  resources  available  to  meet  them.  Focus 
should  be  on  understanding  technological  and  design  implications  and 
achieving  a  high  level  of  technology  maturity  at  the  start  of  system 
development.  This  means  that  the  critical  technologies  needed  to  meet 
essential  product  requirements  must  be  demonstrated  to  work  in  their 
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intended  environment.  The  technology  readiness  level  for  each  critical 
technology  is  the  metric  we  use  to  measure  technology  maturity.1 


•  Knowledge  point  2,  evidenced  by  the  development  of  engineering 

prototypes  and  the  completion  of  engineering  drawings  for  an  integrated 
product  at  the  system  design  review.  This  metric  provides  tangible 
evidence  that  the  product’s  design  is  stable,  meaning  it  has  a  high 
probability  of  meeting  customer  requirements,  as  well  as  cost,  schedule, 
and  reliability  targets.  A  best  practice  is  to  achieve  design  stability  at  the 
system-level  critical  design  review,  usually  held  midway  through 
development.  Completion  of  at  least  90  percent  of  engineering  drawings  is 
the  metric  we  use  to  measure  design  stability. 


•  Knowledge  point  3,  evidenced  by  the  demonstration  that  critical 
manufacturing  processes  are  in  control — that  is,  they  are  repeatable, 
sustainable,  and  capable  of  consistently  producing  parts  within  the 
product’s  quality  tolerances  and  standards — at  the  start  of  production.  One 
hundred  percent  of  critical  manufacturing  processes  in  control  is  the 
metric  we  use  to  evaluate  manufacturing  maturity. 


Figure  1:  Knowledge-Based  Acquisition  Process 
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Metric:  100  percent  of  critical 
manufactoring  processes  in  control 


Source:  GAO  analysis  of  commercial  best  practices. 


Each  of  these  metrics  gauges  the  point  when  the  requisite  level  of 
knowledge  has  been  attained  for  a  product  in  relation  to  where  that 


1  Technology  readiness  levels,  originally  developed  by  the  National  Aeronautics  and  Space 
Administration,  are  measured  on  a  scale  of  1  to  9,  beginning  with  paper  studies  of  a 
technology’s  feasibility  and  culminating  with  a  technology  fully  integrated  into  a  completed 
product.  See  Attachment  2  for  the  definitions  of  technology  readiness  levels. 
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product  is  in  its  development.  World-class  firms  we  have  visited  work  hard 
to  establish  metrics  such  as  these  and  their  decision  makers  are  required 
to  consider  them  before  deciding  to  advance  a  program  to  the  next  level. 
Theses  types  of  metrics  also  help  decision  makers  gauge  progress  in 
meeting  cost,  schedule,  and  performance  goals  and  ensure  that  managers 
will  (1)  conduct  activities  to  capture  relevant  product  development 
knowledge,  (2)  provide  evidence  that  this  knowledge  has  been  captured, 
and  (3)  hold  decision  reviews  to  determine  that  appropriate  knowledge 
has  been  captured  before  moving  the  product  to  the  next  phase.  The  result 
is  a  product  development  process  that  provides  critical  measurements  of 
knowledge,  holds  decision  makers  accountable,  and  delivers  the  expected 
results  in  a  predictable  manner.  Attachment  1  to  this  statement  provides  a 
detailed  list  of  activities  that  would  provide  program  managers  with  the 
requisite  technology,  design,  and  manufacturing  knowledge  at  key  points 
in  time  during  development. 

We  have  used  these  metrics  to  identify  problems  on  major  weapon  system 
acquisition  programs  and  have  found  a  strong  correlation  between  each  of 
them  and  cost  and  schedule  outcomes.  For  example,  for  47  weapon 
programs  in  DOD’s  2008  portfolio  of  major  defense  acquisitions,  we 
assessed  the  knowledge  attained  at  key  decision  points  in  the  acquisition 
process  and  found  the  following:2 

•  Most  programs  have  started  system  development  without  mature 

technologies.  Only  4  of  the  36  programs  that  provided  data  on  technical 
maturity  at  development  start  did  so  with  fully  mature  critical 
technologies.  Further,  only  14  of  39  programs  that  provided  data  have  or 
plan  to  have  demonstrated  all  of  their  technologies  in  a  realistic 
environment  prior  to  system-level  critical  design  review,  at  which  point 
the  system’s  design  should  be  stable.  The  5  newer  programs — those 
initiated  since  20033 — have  higher  levels  of  technology  maturity,  with  all  6 


2We  did  this  by  collecting  data  directly  from  program  offices  using  a  questionnaire.  These 
programs  are  primarily  in  development  and,  therefore,  most  relevant  to  current  decisions 
about  which  programs  should  receive  substantial  investments  of  research  and  development 
funding  now  and  large  amounts  of  procurement  funding  in  the  future.  Defense 
Acquisitions:  Assessment  of  Selected  Weapons  Programs.  GAO-09-326SP.  Washington, 
D.C.:  March  30,  2009. 

3In  2003,  DOD  revised  its  primary  acquisition  policy  to  state  that  technologies  should  be 
demonstrated  in  a  relevant  environment  prior  to  starting  an  acquisition  program.  In  2006, 
this  standard  became  a  statutory  requirement  for  all  major  defense  acquisition  programs  in 
the  National  Defense  Authorization  Act  for  Fiscal  Year  2006,  Pub.  L.  No.  109-163,  §  801, 
codified  at  10  U.S.C.  §  2366b. 
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programs  demonstrating  their  technologies  in  a  relevant  environment 
prior  to  development  start,  in  accordance  with  DOD  and  statutory  criteria. 
However  only  lof  these  programs  met  the  best  practice  standard  of 
demonstrating  critical  technologies  in  an  operational  environment.  Last 
year,  we  determined  that  programs  with  immature  technologies  at  the 
start  of  system  development  experienced  44  percent  higher  cost  growth 
than  programs  that  began  with  mature  technologies. 

•  Programs  that  have  held  design  reviews  in  recent  years  reported  higher 
levels  of  design  knowledge.  However,  designs,  on  average,  are  still  far 
from  stable.  For  the  24  programs  in  our  assessment  that  held  a  critical 
design  review  since  2003,  the  average  percentage  of  total  expected  design 
drawings  releasable  at  this  review  was  65  percent,  compared  to  a  best 
practice  standard  of  90  percent.  We  have  found  that  programs  moving 
forward  into  system  demonstration  with  low  levels  of  design  stability  are 
more  likely  than  other  programs  to  encounter  costly  design  changes  and 
parts  shortages  that,  in  turn,  cause  labor  inefficiencies,  schedule  delays, 
and  quality  problems. 

Attachment  3  represents  our  notional  depiction  of  the  problems  and 
outcomes  that  can  typically  be  expected  when  these  knowledge  metrics 
are  followed  versus  when  they  are  not.  Generally  speaking,  programs  that 
move  forward  without  retiring  technology,  design,  and  manufacturing  risk 
at  appropriate  junctures  will  encounter  a  cascade  of  problems  beginning 
with  design  changes  and  continuing  with  parts  shortages,  changes  to 
manufacturing  processes,  labor  inefficiencies,  and  quality  problems.  All  of 
these  problems  delay  programs  and  add  to  their  development  costs.  We 
have  found,  for  example,  that  a  significant  portion — about  70  percent-of 
the  total  development  cost  growth  in  programs  typically  occurs  after  the 
design  review. 


Outcome  Metrics 
Provide  Insight  into 
the  Health  and 
Performance  of 
Individual  Weapon 
System  Programs  and 
DOD’s  Total  Portfolio 


Program  outcome  metrics — quantitative  measures  of  cost,  schedule,  and 
performance,  and  changes  in  these  factors  over  time — provide  useful 
indicators  of  the  health  of  acquisition  programs  and  facilitate  analyses  of 
how  well  programs  are  meeting  cost,  schedule,  and  performance  goals. 
When  assessed  regularly  for  changes  and  the  reasons  that  cause  changes, 
such  indicators  can  be  valuable  tools  for  improving  insight  and  oversight 
of  individual  programs  as  well  as  DOD’s  total  portfolio  of  major  defense 
acquisitions.  Over  the  years  we  have  reported  cost,  schedule  and 
performance  data — good  and  bad — on  numerous  weapon  systems.  Our 
work  continues  to  identify  systemic  and  program-specific  causes  for  cost, 
schedule,  and  performance  problems  and  has  led  us  to  designate,  since 
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1990,  DOD’s  management  of  major  weapon  system  acquisitions  as  a  high 
risk  area. 

To  improve  acquisition  performance  and  address  the  factors  that  keep 
weapon  acquisitions  on  the  high  risk  list,  DOD  is  working  with  us  and  the 
Office  of  Management  and  Budget  to  develop  a  comprehensive  set  of 
outcome  metrics  to  provide  better,  comprehensive,  and  consistent 
measures  of  program  cost  and  schedule  performance.  Last  year,  this 
cooperative  effort  resulted  in  agreement  to  track  trends  and  changes  in 
programs  from  their  original  baselines,  from  5  years  ago,  and  from  the 
previous  year,  for  the  following  data  points: 

•  Development  cost; 

•  Procurement  cost; 

•  Total  program  cost; 

•  Quantities  to  be  procured; 

•  Procurement  unit  costs; 

•  Total  program  unit  costs; 

•  Cycle  time  from  Milestone  B  to  Initial  Operational  Capability 

DOD  initiated  a  pilot  study  of  7  major  defense  programs  to  assess  the 
adequacy  of  the  proposed  metrics  and  their  value  in  analyzing 
performance,  and  the  results  proved  promising.  DOD  approved  the 
outcome  metrics  and  intends  to  collect  and  report  such  data  on  an  annual 
basis.  Efforts  to  develop  similar  metrics  on  schedule  performance 
continue. 

We  believe  that  the  metrics  DOD  plans  to  use  are  valuable  for  providing 
insight  into  the  performance  of  weapon  system  programs.  We  have  used 
similar  metrics  for  many  years  in  assessing  programs.  For  example,  we 
recently  reported  that  ten  of  DOD’s  largest  acquisition  programs, 
commanding  about  half  the  overall  acquisition  dollars  in  the  department’s 
2008  portfolio  of  major  programs,  have  experienced  significant  cost 
growth  and  have  seen  quantities  reduced  by  almost  a  third  (see  table  1). 
The  two  largest  programs — t  he  Joint  Strike  Fighter  and  the  Future  Combat 
System — represent  significant  cost  risk  moving  forward  and  will  dominate 
the  portfolio  for  years.  Since  these  programs  consume  such  a  large  portion 
of  the  funding  that  DOD  spends  on  research  and  development  and 
procurement,  their  performance  also  affects  other  major  weapon 
acquisitions,  smaller  acquisition  programs,  and  DOD’s  ability  to  fund  and 
acquire  other  supplies  and  equipment  as  well. 
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Table  1 :  Changes  in  Costs  and  Quantities  for  Ten  of  the  Highest  Cost  Acquisition  Programs 


Total  cost 


(fiscal  year  2009  dollars  Acquisition 

in  millions)  Total  quantity  unit  cost 


Program 

First  full 
estimate 

Current 

estimate 

First  full 
estimate 

Current 

estimate 

Percentage 

change 

Joint  Strike  Fighter 

206,410 

244,772 

2,866 

2,456 

38 

Future  Combat  System 

89,776 

129,731 

15 

15 

45 

Virginia  Class  Submarine 

58,378 

81 ,556 

30 

30 

40 

F-22A  Raptor 

88,134 

73,723 

648 

184 

195 

C-17  Globemaster  III 

51,733 

73,571 

210 

190 

57 

V-22  Joint  Services  Advanced  Vertical  Lift  Aircraft 

38,726 

55,544 

913 

458 

186 

F/A-18E/F  Super  Flornet 

78,925 

51 ,787 

1,000 

493 

33 

Trident  II  Missile 

49,939 

49,614 

845 

561 

50 

CVN  21  Nuclear  Aircraft  Class  Carrier 

34,360 

29,914 

3 

3 

-13 

P-8A  Poseidon  Multi-mission  Maritime  Aircraft 

29,974 

29,622 

115 

113 

1 

Source:  GAO  analysis  of  DOD  data. 

While  program  outcome  metrics  are  good  measures  of  individual  program 
performance,  the  collective  performance  of  DOD’s  portfolio  of  major 
defense  acquisition  programs  is  a  key  indicator  of  how  well  the 
department’s  acquisition  system  generates  the  return  on  investment  it 
promises  to  the  warfighter,  Congress,  and  the  taxpayer.  Portfolio  metrics 
also  provide  senior  leaders  and  Congress  with  a  snapshot  of  the 
cumulative  impact  of  current  investment  decisions  and  poor  program 
performance  on  future  budgets.  In  our  annual  assessment  of  selected 
weapon  programs,  we  analyzed  the  performance  of  DOD  programs  at  the 
portfolio  level  by  comparing  programs’  initial  cost,  schedule,  and  quantity 
estimates  to  their  current  estimates,  based  on  data  obtained  from  the 
Selected  Acquisition  Reports.  This  year’s  cumulative  results,  reported 
earlier  this  week, 4  are  shown  in  table  2. 


4  GAO,  Defense  Acquisitions:  Assessments  of  Selected  Weapon  Programs,  GAO-09-326SP 
(Washington,  D.C.:  Mar.  30,  2009). 
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Table  2:  Analysis  of  DOD  Major  Defense  Acquisition  Program  Portfolios3 


Fiscal  year  2009  dollars 

Fiscal  Year 

2003 

2007 

2008 

Portfolio  size 

Number  of  programs 

77 

95 

96 

Total  planned  commitments 

$1.2 

trillion 

$1.6 

trillion 

$1.6 

trillion 

Commitments  outstanding 

$724.2 

billion 

$875.2 

billion 

$786.3 

billion 

Portfolio  indicators 

Change  to  total  RDT&E  costs 
from  first  estimate 

37 

percent 

40 

percent 

42 

percent 

Change  to  total  acquisition  cost 
from  first  estimate 

19 

percent 

26 

percent 

25 

percent 

Total  acquisition  cost  growth 

$183 

billion 

$301.3 

billion" 

$296.4 

billion 

Share  of  programs  with  25 
percent  increase  in  program 
acquisition  unit  cost  growth 

41 

percent 

44 

percent 

42 

percent 

Average  schedule  delay  in 
delivering  initial  capabilities 

18 

months 

21 

months 

22 

months 

Source:  GAO  analysis  of  DOD  data. 

“Data  were  obtained  from  DOD’s  Selected  Acquisition  Reports  (dated  December  2002,  2006,  and 
2007).  In  a  few  cases  data  were  obtained  directly  from  program  offices.  The  number  of  programs 
reflects  the  programs  with  Selected  Acquisition  Reports;  however,  in  our  analysis  we  have  broken  a 
few  Selected  Acquisition  Reports  programs  into  smaller  elements  or  programs.  Not  all  programs  had 
comparable  cost  and  schedule  data  and  these  programs  were  excluded  from  the  analysis  where 
appropriate.  Portfolio  performance  data  do  not  include  costs  of  developing  Missile  Defense  Agency 
elements  or  the  DIMHRS  program. 

“The  acquisition  cost  growth  for  the  2007  portfolio  was  $295  billion  in  2008  constant  dollars. 

Analyzing  the  data  and  comparing  metrics  from  different  time  periods 
provides  unique  insights  into  the  relative  health  of  the  portfolio  and 
trends.  From  2003  to  2008,  the  number  of  programs  in  DOD’s  major 
defense  acquisition  portfolio  has  grown  from  77  to  96.  Total  costs  for  these 
programs  now  total  $1.6  trillion  with  almost  one-half  of  this  amount  still  to 
be  spent.  Outcome  metrics  for  2008  show  worsening  performance  in  all 
categories  compared  to  the  2003  portfolio  and  mixed  performance — some 
better,  some  worse — compared  to  the  2007  data.  While  DOD  is  committing 
substantially  more  investment  dollars  to  developing  and  procuring  new 
weapon  systems,  the  total  acquisition  costs  for  the  2008  portfolio  has 
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grown  by  $296  billion  over  initial  estimates  and  the  average  schedule  delay 
in  delivering  capabilities  to  the  warfighter  averages  22  months. 
Implications  for  the  future  are  obvious.  Continued  cost  growth  reduces 
DOD’s  buying  power  and  results  in  less  funding  being  available  for  other 
DOD  priorities  and  programs.  As  program  costs  increase,  DOD  must 
request  more  funding  to  cover  overruns,  make  trade-offs  with  existing 
programs,  delay  the  start  of  new  programs,  take  funds  from  other 
accounts,  or  reduce  procurement  quantities.  Continued  failure  to  deliver 
weapon  systems  on  time  delays  providing  critical  capabilities  to  the 
warfighter  and  results  in  operating  costly  legacy  systems  longer  than 
expected,  finding  alternatives  to  fill  capability  gaps,  or  going  completely 
without  the  capability. 


Key  Prerequisites 
Need  to  Be  Met  to 
Improve  the  Value  of 
Metrics  and  Achieve 
Better  Acquisition 
Program  Outcomes 


While  the  metrics  discussed  above  can  provide  valuable  knowledge  about 
potential  problems  and  additional  information  on  the  health  of  DOD’s 
acquisition  programs,  metrics  alone  may  not  be  sufficient  if  the 
department  does  not  do  a  better  job  ensuring  that  acquisitions  begin  with 
realistic  plans  and  baseline  estimates  for  cost  and  schedules  prior  to 
development  start.  We  believe  there  is  a  clear  set  of  prerequisites  that 
must  be  a  part  of  any  acquisition  strategy  before  any  measurement  of  the 
acquisition’s  health  can  be  valuable.  Otherwise,  metrics  measured  against 
unrealistic  estimates  will  do  no  good.  These  key  prerequisites  for 
obtaining  realistic  baselines  include: 


•  Establishing  a  clear,  knowledge-based,  evolutionary  business  case  for  the 
product.  This  business  case  must:  validate  that  a  need  exists;  determine 
that  resources  are  available  to  develop  a  product  that  will  meet  the  need; 
determine  that  the  product  developer  has  a  knowledge-based  plan  and 
strategy  to  deliver  the  product;  establish  reasonable  estimates  for  cost, 
delivery  time  and  quantities;  and  ensure  available  funding  for  the  product. 
All  of  these  elements  of  the  business  case  should  also  be  agreed  upon  by 
major  stakeholders  across  the  requirements,  funding,  acquisition,  and 
warfighting  communities. 

•  Separating  technology  development  activities  from  product  development 
activities.  The  process  of  developing  technology  culminates  in 
discovery — the  gathering  of  knowledge — and  must,  by  its  nature,  allow 
room  for  unexpected  results  and  delays.  Leading  firms  do  not  ask  their 
product  managers  to  develop  technology  because  they  have  learned  the 
hard  way  that  invention  cannot  be  scheduled.  When  immature 
technologies  are  brought  onto  the  critical  path  of  product  development 
programs  too  early,  they  often  cause  long  delays  in  an  environment  where 
large  workforces  must  be  employed,  complex  tools,  plants,  and  facilities 
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must  be  operated,  long  and  expensive  supplier  networks  must  be  paid,  and 
the  product  itself  must  sometimes  be  redesigned  once  the  final  form  of  the 
technologies  is  known.  Successful  programs  give  responsibility  for 
maturing  technologies  to  science  and  technology  organizations,  rather 
than  the  program  or  product  development  managers,  because  the  science 
and  technology  environment  is  less  costly.  We  have  recommended  in  the 
past  that  DOD’s  risks  should  be  taken  in  the  science  and  technology  arena 
and  that  more  funding  should  be  made  available  to  this  process  to  do  so. 

•  Limiting  time  and  requirements  for  product  development  to  manageable 
levels.  Product  developers  should  strive  to  deliver  the  best  available 
capabilities  within  realistic  timeframes  and  should  expect  to  continue  to 
develop  new  capabilities  when  they  are  technologically  feasible.  By 
limiting  product  development  cycle  times  to  6  years  or  less,  DOD  could 
assimilate  new  technologies  into  weapon  systems  more  frequently, 
accelerate  delivery  of  new  technology  to  the  warfighter,  hold  program 
managers  accountable,  and  make  more  frequent  and  predictable  work  in 
production,  where  contractors  and  the  industrial  base  can  profit  by  being 
efficient.  Too  many  major  acquisitions  currently  take  the  opposite 
approach  by  seeking  to  deliver  a  revolutionary  “big  bang”  capability  in  one 
step.  This  means  that  programs  are  more  risky,  delivery  takes  as  long  as  15 
years  in  some  cases,  and  costs  grow  at  exponential  rates  from  the  original 
baseline  due  to  the  risky  nature  of  the  acquisition  strategy.  We  point  to  the 
private  sector  and  some  past  defense  acquisitions,  such  as  the  F-16 
program,  as  models  for  this  practice. 

•  Employing  early  systems  engineering  discipline  in  order  to  develop 
realistic  cost  and  schedule  estimates  prior  to  development  start.  Early 
systems  engineering  provides  the  knowledge  a  product  developer  needs  to 
identify  and  resolve  performance  and  resource  gaps  before  product 
development  begins  either  by  reducing  requirements,  deferring  them  to 
the  future,  or  increasing  the  estimated  cost  for  the  weapon  system’s 
development.  Requirements  that  are  too  risky  given  the  state  of 
technology  and  design  should  not  be  allowed  into  this  expensive 
environment. 

•  Making  a  commitment  to  fully  fund  programs  once  they  are  approved. 
This  would  require  the  department  to  ensure  that  it  does  not  have  too 
many  programs  underway  given  the  amount  of  available  resources. 

•  Setting  priorities  from  the  top  to  ensure  tha  t  candida  te  programs  are 
truly  needed  and  have  a  solid  plan  for  delivery.  DOD  will  continue  to 
experience  poor  acquisition  outcomes  until  it  begins  making  choices  that 
reflect  the  most  important  needs  of  the  joint  warfighter  and  match 
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requirements  with  available  resources.  The  urge  to  accept  all  candidate 
programs  and  to  go  for  the  “big  bang”  capability  without  the  knowledge  to 
achieve  it  should  be  resisted.  Only  the  best  candidates-defined  in  terms  of 
priorities,  resource  availability,  and  executability-should  be  approved. 

There  is  no  doubt  that  the  current  state  of  the  department’s  acquisition 
process  is  too  expensive  for  the  taxpayer  and  not  timely  enough  for  the 
warfighter.  The  following  illustration  reinforces  this  point. 


Figure  2:  Cost  Remaining  Versus  Annual  Appropriations  for  Major  Defense  Acquisitions 

Dollars  (in  billions) 


Fiscal  year 

Costs  Remaining  for  Major  Defense  Acquisitions 
Annual  RDTE  and  Procurement  Appropriations 
Source:  DOD  (data):  GAO  (analysis  and  presentation). 


This  figure  depicts  an  investment  strategy  for  major  weapon  systems  that 
continues  to  increase  the  costs  to  develop  our  existing  weapons  well  into 
the  future  while  the  funding  available  to  retire  those  costs  appears  capped 
at  a  very  low  level.  While  costs  continue  to  rise  as  the  result  of  more  and 
more  risky  programs  being  added  to  the  portfolio,  our  ability  to  allocate 
funds  for  these  costs  appears  to  be,  at  best,  capped  at  very  low 
percentages  of  the  total  cost.  We  could  measure  the  risk  of  these 
acquisitions  much  better  than  we  have  in  the  past  if  we  set  the  appropriate 
prerequisites  for  their  initiation,  measure  the  knowledge  that  must  be  in 
place  at  various  points,  and  continue  to  monitor  their  health  in  terms  of 
cost  and  schedule. 
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Concluding  Remarks 

Measuring  the  performance  of  weapon  system  programs  both  individually 
and  collectively  is  critical  for  determining  whether  the  warfighter  and  the 
taxpayer  are  receiving  the  promised  return  on  investment.  No  single 
metric,  however,  can  capture  the  whole  picture  of  how  well  programs  are 
performing.  It  is  important  to  look  at  knowledge  and  outcome  metrics. 
Knowledge  metrics  provide  key  information  for  determining  whether 
programs  have  the  requisite  knowledge  to  move  from  one  phase  of 
development  to  the  next  and  are  at  risk  of  cost  and  schedule  overruns. 
Outcomes  metrics  are  also  needed  to  provide  temperature  checks  on  the 
health  and  status  of  individual  programs  and  the  portfolio  of  programs  as  a 
whole.  These  metrics  are  vital  for  informing  program  decision  making  and 
helping  to  manage  programs. 

Metrics  by  themselves  do  not  solve  problematic  acquisitions.  Ultimately, 
DOD  still  needs  to  do  a  better  job  planning  and  executing  programs  to 
achieve  better  outcomes.  Critical  to  achieving  successful  outcomes  is 
establishing  knowledge-based,  realistic  program  baselines.  Without 
realistic  baselines,  there  is  no  foundation  for  accurately  measuring  the 
knowledge  and  health  of  programs.  Over  the  past  several  years,  our  work 
has  highlighted  a  number  of  underlying  causes  for  why  DOD  does  not 
effectively  manage  the  acquisition  of  weapon  system  programs.  DOD 
recently  revised  its  acquisition  policy  to  provide  a  better  foundation  for 
developing  weapon  systems,  however,  reform  will  not  be  achieved  without 
fundamental  changes  to  the  overall  acquisition  culture  and  environment 
that  exists  in  DOD.  I  would  be  pleased  to  discuss  these  causes  and  issues 
with  the  Committee  at  a  future  time. 

Mr.  Chairman,  this  concludes  my  prepared  statement.  I  would  be  happy  to 
answer  any  questions  you  may  have  at  this  time. 
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Attachment  1:  Knowledge-Based  Activities 


Knowledge  Point  1 :  Start  of  product  development  activities 

Best  practice  metric:  Technology  readiness  level  7  (indicating  technologies  work  in  an  operational  environment) 

Demonstrate  technologies  to  high  readiness  levels 

Ensure  that  requirements  for  the  product  increment  are  informed  by  preliminary  design  using  systems  engineering  process  (such  as 
prototyping  of  preliminary  design) 

Establish  cost  and  schedule  estimates  for  product  on  the  basis  of  knowledge  from  preliminary  design  using  system  engineering  tools 
(such  as  prototyping  of  preliminary  design) 

Constrain  development  phase  (5  to  6  years  or  less)  for  incremental  development 
Ensure  development  phase  fully  funded  (programmed  in  anticipation  of  milestone) 

Align  program  manager  tenure  to  complete  development  phase 

Contract  strategy  that  separates  system  integration  and  system  demonstration  activities 

Conduct  independent  cost  estimate 

Conduct  independent  program  assessment 

Conduct  major  milestone  decision  review  for  development  start 

Knowledge  Point  2:  Transition  from  system  integration  to  system  demonstration  activities 
Best  practice  metric:  90  percent  of  design  drawings  are  complete  by  the  critical  design  review 

Complete  system  critical  design  review 

Complete  90  percent  of  engineering  design  drawing  packages 

Complete  subsystem  and  system  design  reviews 

Demonstrate  with  system  integration  prototype  that  design  meets  requirements 
Complete  the  failure  modes  and  effects  analysis 
Identify  key  system  characteristics 
Identify  critical  manufacturing  processes 

Establish  reliability  targets  and  growth  plan  on  the  basis  of  demonstrated  reliability  rates  of  components  and  subsystems 
Conduct  independent  cost  estimate 
Conduct  independent  program  assessment 

Conduct  major  milestone  decision  review  to  enter  system  demonstration 

Knowledge  Point  3:  Initiation  of  producing  a  product  to  be  delivered  to  customer 
Best  practice  metric:  100  percent  of  critical  manufacturing  processes  are  in  control 

Demonstrate  manufacturing  processes 

Build  and  test  production-representative  prototypes  to  demonstrate  product  in  operational  environment 
Test  production-representative  prototypes  to  achieve  reliability  goal 
Collect  statistical  process  control  data 

Demonstrate  that  critical  processes  are  capable  and  in  statistical  control 
Independent  cost  estimate 
Independent  program  assessment 

Conduct  major  milestone  decision  review  to  begin  production 

Source:  GAO  analysis  of  commercial  best  practices. 
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Attachment  2:  Technology  Readiness  Levels 


Technology  readiness 
level 

Description 

Hardware 

Software 

Demonstration 

Environment 

1.  Basic  principles 
observed  and 
reported. 

Lowest  level  of  technology  readiness. 
Scientific  research  begins  to  be  translated 
into  applied  research  and  development. 
Examples  might  include  paper  studies  of  a 
technology's  basic  properties 

None  (paper  studies  and 
analysis) 

None 

2.  Technology  concept 
and/or  application 
formulated. 

Invention  begins.  Once  basic  principles  are 
observed,  practical  applications  can  be 
invented.  The  application  is  speculative 
and  there  is  no  proof  or  detailed  analysis  to 
support  the  assumption.  Examples  are  still 
limited  to  paper  studies. 

None  (paper  studies  and 
analysis) 

None 

3.  Analytical  and 

experimental  critical 
function  and/or 
characteristic  proof 
of  concept. 

Active  research  and  development  is 
initiated.  This  includes  analytical  studies 
and  laboratory  studies  to  physically 
validate  analytical  predictions  of  separate 
elements  of  the  technology.  Examples 
include  components  that  are  not  yet 
integrated  or  representative. 

Analytical  studies  and 
demonstration  of  nonscale 
individual  components  (pieces  of 
subsystem). 

Lab 

4.  Component  and/or 
breadboard. 
Validation  in 
laboratory 
environment. 

Basic  technological  components  are 
integrated  to  establish  that  the  pieces  will 
work  together.  This  is  relatively  “low 
fidelity”  compared  to  the  eventual  system. 
Examples  include  integration  of  “ad  hoc” 
hardware  in  a  laboratory. 

Low  fidelity  breadboard. 

Integration  of  nonscale 
components  to  show  pieces  will 
work  together.  Not  fully  functional 
or  form  or  fit  but  representative  of 
technically  feasible  approach 
suitable  for  flight  articles. 

Lab 

5.  Component  and/or 
breadboard 
validation  in  relevant 
environment. 

Fidelity  of  breadboard  technology 
increases  significantly.  The  basic 
technological  components  are  integrated 
with  reasonably  realistic  supporting 
elements  so  that  the  technology  can  be 
tested  in  a  simulated  environment. 

Examples  include  “high  fidelity”  laboratory 
integration  of  components. 

High  fidelity  breadboard. 

Functionally  equivalent  but  not 
necessarily  form  and/or  fit  (size 
weight,  materials,  etc).  Should  be 
approaching  appropriate  scale. 
May  include  integration  of  several 
components  with  reasonably 
realistic  support 
elements/subsystems  to 
demonstrate  functionality. 

Lab  demonstrating 
functionality  but  not  form 
and  fit.  May  include  flight 
demonstrating  breadboard 
in  surrogate  aircraft. 
Technology  ready  for 
detailed  design  studies. 

6.  System/subsystem 
model  or  prototype 
demonstration  in  a 
relevant 
environment. 

Representative  model  or  prototype  system, 
which  is  well  beyond  the  breadboard  tested 
for  TRL  5,  is  tested  in  a  relevant 
environment.  Represents  a  major  step  up 
in  a  technology’s  demonstrated  readiness. 
Examples  include  testing  a  prototype  in  a 
high  fidelity  laboratory  environment  or  in 
simulated  realistic  environment. 

Prototype. 

Should  be  very  close  to  form,  fit 
and  function.  Probably  includes 
the  integration  of  many  new 
components  and  realistic 
supporting  elements/subsystems 
if  needed  to  demonstrate  full 
functionality  of  the  subsystem. 

High-fidelity  lab 
demonstration  or 
limited/restricted  flight 
demonstration  for  a 
relevant  environment. 
Integration  of  technology 
is  well  defined. 

Page  14 


GAO-09-543T 


Technology  readiness 
level 

Description 

Hardware 

Software 

Demonstration 

Environment 

7.  System  prototype 
demonstration  in  a 
realistic 
environment. 

Prototype  near  or  at  planned  operational 
system.  Represents  a  major  step  up  from 
TRL  6,  requiring  the  demonstration  of  an 
actual  system  prototype  in  a  realistic 
environment,  such  as  in  an  aircraft,  vehicle 
or  space.  Examples  include  testing  the 
prototype  in  a  test  bed  aircraft. 

Prototype.  Should  be  form,  fit  and 
function  integrated  with  other  key 
supporting  elements/subsystems 
to  demonstrate  full  functionality  of 
subsystem. 

Flight  demonstration  in 
representative  realistic 
environment  such  as 
flying  test  bed  or 
demonstrator  aircraft. 
Technology  is  well 
substantiated  with  test 
data. 

8.  Actual  system 

completed  and  “flight 
qualified”  through 
test  and 
demonstration. 

Technology  has  been  proven  to  work  in  its 
final  form  and  under  expected  conditions. 

In  almost  all  cases,  this  TRL  represents  the 
end  of  true  system  development.  Examples 
include  developmental  test  and  evaluation 
of  the  system  in  its  intended  weapon 
system  to  determine  if  it  meets  design 
specifications. 

Flight  qualified  hardware 

Developmental  Test  and 
Evaluation  (DT&E)  in  the 
actual  system  application 

9.  Actual  system  “flight  Actual  application  of  the  technology  in  its  Actual  system  in  final  form  Operational  Test  and 

proven”  through  final  form  and  under  mission  conditions,  Evaluation  (OT&E)  in 

successful  mission  such  as  those  encountered  in  operational  operational  mission 

operations.  test  and  evaluation.  In  almost  all  cases,  conditions 

this  is  the  end  of  the  last  “bug  fixing” 
aspects  of  true  system  development. 

Examples  include  using  the  system  under 


operational  mission  conditions. 


Source:  GAO  and  its  analysis  of  National  Aeronautics  and  Space  Administration  data. 
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Attachment  3:  Notional  Illustration  Showing 
the  Different  Paths  That  a  Product’s 
Development  Can  Take 
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